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ABSTRACT 


The  desire  for  direct  interaction  between  man  and 
machine  has  led  to  the  study  of  computer  interpretation 
of  free-hand  motions  of  a  stylus  and  the  "real-time" 
responses  to  these  motions.  An  operating  environment 
is  discussed,  utilizing  elements  of  pictorial  and  verbal 
languages . 
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ON  THE  DEVELOPMENT  OF  EQUITABLE  GRAPHIC  I/O 
T.  0.  Ellis  and  W.  L.  Sibley* 

The  RAND  Corporation,  Santa  Monica,  California 


Using  the  broad  capabilities  of  modern  displays, 
information — text ,  drawings,  and  even  motion— can  be 
structured  so  that  a  computer  can  communicate  it  to  man 
in  a  manner  which  is  direct  and  natural  to  the  man. 
However,  this  communication  coin  has  two  sides.  The 
same  descriptive  powers  should  exist  for  input  as  for 
output.  Or,  ideally,  if  the  display  is  considered  as  a 
common  working  area,  then  it  might  suffice  as  the  com¬ 
plete  communication  medium.  The  user  should  be  able  to 
create  and  manipulate  the  contents  of  the  display  in  ways 
which  are,  again,  directly  meaningful  and  natural  to  him. 
In  many  cases,  only  Graphic  I/O  systems  are  sufficiently 
two-dimensional  and  dynamic  to  permit  this  directness. 


Any  views  expressed  in  this  Paper  are  those  of  the 
authors.  They  should  not  be  interpreted  as  reflecting 
the  views  of  The  RAND  Corporation  or  the  official  opinion 
or  policy  of  any  of  its  governmental  or  private  research 
sponsors.  Papers  are  reproduced  by  The  RAND  Corporation 
as  a  courtesy  to  members  of  its  staff. 

This  Paper  was  prepared  for  publication  in  IEEE 
Transactions  on  Human  Factors  in  Electronics. 
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Constder  the  single  problem  of  inserting  a  word  into 
a  line  of  text.  With  an  ordinary  typewriter,  commands 
must  be  given  to  an  intermediary,  as  follows:  "In  line 
7  before  the  third  word,  insert  word  x. (or  some 
suitable  encoding  of  that  statement).  Whereas,  if  the 
text  appears  as  a  dynamic  picture  on  a  display  then, 
with  appropriate  controls,  the  user  may  insert  the  word 
directly  as  easily  as  on  a  paper  draft.  However,  these 
actions— the  elements  of  a  graphic-  input  language- -must 
be  interpretable  by  the  machine  if  they  are  to  be  opera¬ 
tionally  meaningful. 

Behind  the  development  of  graphic  language  elements 
are  two  major  issues: 

1)  The  hardware  facilities  which  allow  a  user  to 
express  himself  freely  in  a  language  both  natural 
to  himself  and  compatible  with  output  capabilities. 

2)  The  interpretive  capabilities  which  make  these 
expressions  meaningful. 

Much  previous  research  is  applicable  to  these  needs. 
Recognition  of  handwritten  symbols  [1-3]  has  been  reason¬ 
ably  successful.  The  "Sketchpad"  effort  [4]  and  its 
successors  developed  ways  to  structure  pictorial  informa¬ 
tion  so  that  contextual  meaning  can  be  given  to  external 
actions  on  the  pictorial  information.  Recently  developed 
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hardware  now  accepts  free-hand  input,  providing  high- 
detail  data  compatible  with  output  capabilities  [5]. 

The  present  state-of-the-art  is  such  that  we  can 
begin  to  refine  these  techniques  toward  their  aggregation 
within  one  operational  system.  Such  an  experimental  system 
(called  GRAIL— Graphic  Input  Language)  is  under  development 

•ft 

at  The  RAND  Corporation.  This  Paper  reports  only  the 
nature  of  this  project  with  respect  to  man-machine  communica 
tions  and  some  of  the  related  experiences.  It  is  too  early 
to  state  measurements  or  conclusions. 

Figure  1  shows  the  console  configuration.  It  Includes 
a  vertical  cathode  ray  tube  (CRT)  display  for  output  and  a 
horizontal  Tablet  with  pen  for  communicating  free-hand 
motions  to  a  computer.  The  computer  controls  the  informa¬ 
tion  on  the  display  surface  except  for  one  spot  of  light. 
This  spot  follows  the  relative  pen  position  on  the  Tablet 
at  all  times.  Conceptually,  this  spot  on  the  CRT  is  the 
pen  point.  A  switch  in  the  pen  tip  senses  downward  pres¬ 
sure  applied  to  the  pen;  such  action  signals  the  computer 
program  to  accept  the  pen  track  data  and  interpret  it  in 
the  context  of  the  information  currently  being  displayed. 

•Hf 

The  GRAIL  project  is  supported  by  the  Advanced 
Research  Projects  Agency. 
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Thus,  the  user  can  describe  position,  symbols,  shapes, 
or  motion  directly  in  place,  as  if  he  were  actually  writing 
on  the  surface  of  the  CRT.  Hence,  the  user's  eyes  are 
always  focused  on  the  CRT  which  is  his  "working  surface." 

When  adding  or  changing  a  symbol  in  the  displayed  picture, 

i 

he  draws  or  prints  that  symbol  by  applying  normal  writing 
pressure  during  movement  to  define  the  separate  strokes 
within  the  symbol.  The  immediate  computer  interpretation 
represents  the  path  as  a  visible  track  on  the  CRT.  This 
first  feedback  not  only  provides  the  "ink"  necessary  to 
the  user's  ability  to  draw,  but  also  gives  the  user  real¬ 
time  indication  of  exactly  what  is  being  interpreted. 

When  this  track  has  described  an  interpretable  symbol,  it 
is  replaced  (in  place)  by  a  normalized  symbol.  Again, 
the  user  can  immediately  and  directly  compare  his  inten¬ 
tions  and  the  computer's  interpretation.  A  major  project 
task  is  to  provide  this  kind  of  feedback  at  each  level 
of  interpretation. 

The  degree  of  directness  of  interaction  attainable 

Sepends  entirely  upon  the  sophistication  of  the 

ses  available  to  interpret  relatively  natural  lan- 
elements  and  relate  them  to  the  problem  context  in 
real  t\ae  (user  time) .  Processes  have  been  developed 


which  allow  flowchart  pictures  to  be  constructed  and 
manipulated  with  a  fixed  set  of  recognizable  symbols. 

Thesa  symbols  may  be  drawn,  at  the  user's  pace,  in  the 
desired  position  and  size.  When  the  pen  is  lifted  at 
the  end  of  each  symbol  description,  the  computer  immed¬ 
iately  replaces  that  free-hand  track  display  with  a 
normalized  or  drafted  figure  whose  type,  size,  and  posi¬ 
tion  are  the  result  of  the  analysis  of  that  track  and  its 
logical  implications  in  the  current  display.  Picture- 
editing  features  include  the  abilities  to  move,  stretch, 
or  erase  elements  of  the  picture — again,  entirely  through 
pen  motion. 

In  its  current  state,  the  symbol-recognition  scheme, 

though  imperfect,  is  productive  and  constantly  being 
* 

improved.  The  recognizable  symbol  set  includes,  in 
addition  to  the  flowchart  figures,  the  upper-case  Latin 
alphabet,  Arabic  numbers,  the  symbols  +,  -,  =,  /,  (,  ), 

1$  *,  $,  .  ,  , ,  ' ,  and  the  text-editing  marks  >,  A, 

and  "scrub"  (erasure).  Again,  when  a  symbol  is  recognized, 
a  "hardware"-generated  symbol  replaces  the  user's  inked 
version.  Since  the  user  may  change  an  existing  symbol  at 

ic 

Reference  6  describes  and  gives  early  -est  results 
of  this  recognition  method. 
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any  time  by  writing  over  it,  and  can  insert  or  delete 
easily  with  the  above  editing  marks,  his  tolerance  of 
recognition  errors  is  generally  high.  Figure  2  indicates 
system  tolerance  of  user  variations.  Text-editing  pro¬ 
cesses  allow  the  above  alphanumerics  to  be  used  as  either 
designations  or  comments  in  the  flowchart  drawings  as 
well  as  in  programming  code  forms. 

Depending  on  the  task  requirements,  appropriate 
display  responses  to  pointing,  printing,  sketching, 
bounding,  dragging,  stretching,  inserting,  erasing,  etc., 
must  occur  in  real  time  and  be  directed  in  a  way  natural 
to  the  user.  A  language  encompassing  these  actions  must 
be  carefully  structured  to  eliminate  ambiguities  in 
interpretation  and  yet  be  so  natural  that  the  user  is 
rarely  uncertain  how  to  accomplish  a  task.  This  natural¬ 
ness  and  directness  allows  the  user  to  think  continuously 
about  his  problem,  rather  than  concern  himself  with  the 
mechanics  of  cooxnunication. 

The  large  amount  of  research  on  automated  design 
will  depend  increasingly  on  such  Intimate  contact  with 
machines.  Whether  the  computer  system  is  oriented  toward 
the  design  of  automobiles,  bridges,  mathematical  models, 
or  computer  programs,  more  and  more  is  being  demanded  of 


3  3  53 
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Fig.  2 — Some  Tracks  Recognized  as  the  Symbol  "3" 
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the  man-machine  communication  channel  to  match  the  com¬ 
puter's  growing  ability  to  quickly  answer  questions  or 
obey  directives. 
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